Skip to content

Add source-aware acceptance eval fixtures - #80

Closed
catoncat wants to merge 1 commit into
agent/issue-71-query-profilefrom
agent/issue-73-source-aware-eval
Closed

Add source-aware acceptance eval fixtures#80
catoncat wants to merge 1 commit into
agent/issue-71-query-profilefrom
agent/issue-73-source-aware-eval

Conversation

@catoncat

@catoncat catoncat commented Jun 26, 2026

Copy link
Copy Markdown
Owner

Refs #73

Scope

Add the source-aware acceptance-eval slice for the experimental public source adapter hardening work.

This PR extends the synthetic acceptance gate so it covers:

  • Codex existing hard fixtures, unchanged as the compatibility path.
  • Claude Code source-qualified find -> read-range for message-level evidence.
  • Pi source-qualified session-level compaction find -> read-page for page context.

Implementation summary

  • eval/acceptance-gate.ts now writes a fixture bundle with per-source roots for codex, claude-code, and pi.
  • runAcceptanceGate() syncs all three roots into the same temporary DB and returns sourceRoots plus sourceSyncs, while preserving the existing fixtureRoot and sync Codex fields.
  • Added hard goldens:
    • claude-code-message-range-context
    • pi-session-page-context
  • eval/acceptance-gate.test.ts now asserts Codex + Claude Code + Pi sync counts and 6/6 hard-case pass coverage.

Verification

  • npx vitest run eval/acceptance-gate.test.ts -> 1 file passed, 1 test passed.
  • npm run eval:acceptance -> 6 hard cases, 6 pass, 0 fail; Codex added 6, Claude Code added 1, Pi added 1.
  • npm run check -> tsc --noEmit plus Vitest, 31 files passed, 206 tests passed.
  • git diff --check -- eval/acceptance-gate.ts eval/acceptance-gate.test.ts -> passed.

Risk

Low. This changes eval/test code only and does not modify production parser projection, source adapter acceptance policy, SQLite schema, or CLI behavior.

Base branch / dependencies

Base branch: agent/issue-71-query-profile

Depends on:

Related but not blocking:

Suggested merge order:

  1. Add evidence-level acceptance eval gate #75
  2. Make query profile scoring affect ranking #79
  3. this PR

Trace

request_id: cxs_continue_roadmap_to_completion_20260626T054330Z_api_append


Summary by cubic

Make the acceptance eval source-aware by writing per-source fixtures and syncing codex, claude-code, and pi into one DB. Adds hard goldens for Claude Code message-range and Pi page-context to meet #73.

  • New Features
    • Per-source fixture roots and syncs: returns sourceRoots and sourceSyncs while keeping fixtureRoot/sync for codex.
    • New hard cases: claude-code-message-range-context and pi-session-page-context.
    • Source-qualified find selectors with context modes: claude-code uses read-range, pi uses read-page.
    • Tests assert per-source sync counts and 6/6 hard-case passes.

Written for commit 1c97c73. Summary will update on new commits.

Review in cubic

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 2 files

Re-trigger cubic

@catoncat

Copy link
Copy Markdown
Owner Author

Superseded by #82, which merged the source-aware acceptance eval fixture work into main after the conditional #71 scoring cleanup.

@catoncat catoncat closed this Jun 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant